Added i18n component and related scripts #1082

yihao03 · 2025-03-28T02:44:22Z

run yarn trans to start translation

…plits the text into manageable chunk

…output path

…e function

…xt handling and error reporting

…toring segment handling to use arrays for better management

…ling in Promise.all

…ity and consistency

… merging logic and remove unnecessary debug logs

…e TRANSLATE tags

…guages in xml

…ager version in package.json

…eadability

… file path usage

…lations inside SCHEME and SCHEMEINLINE tags

…ded try catch when parsing xml to json to report failed parsing possibly attributed to unsound xml structure.

openai api

- Created a new XML file for references (97references97.xml) containing a comprehensive list of references used in the SICP JS project. - Added a new XML file for the index preface (98indexpreface98.xml) to provide context and formatting for the index section. - Introduced a new XML file for the making section (99making99.xml) detailing the background, interactive features, and development history of the SICP JS project. - Updated subsection2.xml to close the previously open SUBSECTION tag and include a comment for clarity.

…LINE tags in XML parsing

yihao03 · 2025-04-12T07:10:47Z

breaking changes are made: xml repositories are divided into folders, currently consisting of en and cn folders to store translated content. The same applies to the json folder after running "yarn json". Frontend needs to be changed accordingly to fetch json files from the corresponding url

…logic

…reaking search data

arnav-goel10 · 2025-05-11T07:44:00Z

i18n/controllers/translator.ts

+  baseURL: process.env.AI_BASEURL
+});
+
+const MAXLEN = Number(process.env.MAX_LEN) || 3000;


The MAXLEN constant should be imported from the config.ts file instead of being redefined here. The config.ts already defines max_chunk_len for the same purpose with the same default value (3000).

Having configuration values defined in multiple places leads to maintenance issues and potential inconsistencies if one location is updated but not the other.

Suggestion: Replace with import { max_chunk_len } from "../config"; and use that value directly.

arnav-goel10 · 2025-05-11T07:57:14Z

i18n/controllers/loggers.ts

+        console.log(
+            `Summary log saved to logs/translation-summary-${timestamp}.log`
+        );


The log path is constructed differently in the code versus the message displayed to users:

// Actual path construction: const logPath = path.join( logDir, `${translationSummaryPrefix}-${timestamp}.log` ); // But message shows hardcoded format: console.log( `Summary log saved to logs/translation-summary-${timestamp}.log` );

arnav-goel10 · 2025-05-11T08:03:57Z

i18n/controllers/parsers.ts

+import { Readable } from "stream";
+import { ignoredTags, max_chunk_len } from "../config";
+import fs, { PathLike } from "fs";
+import { FileLike } from "openai/uploads.mjs";


The import import { FileLike } from "openai/uploads.mjs"; appears to be unused in this file. There are no references to the FileLike type throughout the code.

arnav-goel10 · 2025-05-11T08:18:47Z

i18n/controllers/parsers.ts

+import fs, { PathLike } from "fs";
+import { FileLike } from "openai/uploads.mjs";
+
+const MAXLEN = max_chunk_len;


The code creates an unnecessary alias for max_chunk_len. Additionally, throughout the code there are instances of Number(MAXLEN) which is redundant since max_chunk_len should already be a Number type in the config file.

arnav-goel10 · 2025-05-11T08:54:45Z

i18n/initializers/initialize.ts

+import fs from "fs";
+import OpenAI from "openai";
+import path, { dirname } from "path";
+import Stream from "stream";


The Stream module is imported but never used.

translated section2 cn xmls

arnav-goel10

LGTM!

RichDom2185

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

xml/en/chapter1/section1/subsection3.xml

xml/en/chapter3/section3/subsection4.xml

xml/en/chapter3/section3/subsection5.xml

xml/en/chapter3/section4/subsection2.xml

xml/en/chapter2/section3/subsection2.xml

RichDom2185 · 2025-07-05T04:15:59Z

.github/workflows/deploy-pages.yml

+      - name: Clone translated_xmls
+        run: |
+          git clone -b translated_xmls https://github.com/source-academy/sicp.git translated_xmls
+          mv translated_xmls/* xml/
+          rm -r translated_xmls


Why are we cloning from a specific branch instead of merging it to the default branch?

RichDom2185 · 2025-07-05T04:16:51Z

.github/workflows/translate-changed.yml

Could you explain what these new workflows are for?

This workflow is for triggering translations for all those English XMLs that are changed. The workflow is not set to be triggered on pushes yet because the AI translation is not very stable yet.

the translate-everything workflow is for translating all English XMLs

RichDom2185 · 2025-07-05T04:17:43Z

.github/workflows/translate-changed.yml

+          echo API_KEY=${{ secrets.OPENAI_KEY }} >> .env
+          # echo API_KEY=${{ secrets.OPENAI_KEY2 }} >> .env


If they are no longer used, the secret should be deleted.

In the start of the project, me and Yi Hao were both given an API key, so there are two secrets. Now one is used as a backup.

RichDom2185 · 2025-07-05T04:18:22Z

.github/workflows/translate-everything.yml

+          echo API_KEY=${{ secrets.OPENAI_KEY }} >> .env
+          # echo API_KEY=${{ secrets.OPENAI_KEY2 }} >> .env


coder114514 · 2025-07-05T11:25:47Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

RichDom2185 · 2025-07-05T11:30:35Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

How is the zh_CN folder different from the translated_xmls branch? Why not just merge the translations directly to master instead of using a separate branch?

coder114514 · 2025-07-05T11:35:12Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

How is the zh_CN folder different from the translated_xmls branch? Why not just merge the translations directly to master instead of using a separate branch?

They are the same. The translated_xmls branch is separately created because in the workflow for translation (translate-changed, translate-everything) an action called deploy which pushes files to a branch is used to push AI generated translations to GitHub, so the branch is created for this.

RichDom2185 · 2025-07-05T11:52:20Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

How is the zh_CN folder different from the translated_xmls branch? Why not just merge the translations directly to master instead of using a separate branch?

They are the same. The translated_xmls branch is separately created because in the workflow for translation (translate-changed, translate-everything) an action called deploy which pushes files to a branch is used to push AI generated translations to GitHub, so the branch is created for this.

I see, noted, and what is the rationale of running it in a workflow, as opposed to running it locally?

coder114514 · 2025-07-05T12:00:12Z

Did not look at the contents of the zh_CN folder. I assume they are generated by AI? Or was there manual editing? If the former, please add a gitattributes file so that reviewers (and GitHub) knows it's machine generated

All the AI translated XMLs will go into the translated_xmls branch, so I will delete the zh_CN folder. Should I add gitattributes file to the translated_xmls branch to mark they are generated by AI?

How is the zh_CN folder different from the translated_xmls branch? Why not just merge the translations directly to master instead of using a separate branch?

They are the same. The translated_xmls branch is separately created because in the workflow for translation (translate-changed, translate-everything) an action called deploy which pushes files to a branch is used to push AI generated translations to GitHub, so the branch is created for this.

I see, noted, and what is the rationale of running it in a workflow, as opposed to running it locally?

sicp.sourceacademy.org is a static site, so the translated XMLs need to be deployed too, and thus we are storing them on GitHub.

This is not really about the translation workflows. The translator can be run locally and then the generated content can be manually pushed to the translated_xmls branch.

yihao03 and others added 27 commits March 12, 2025 22:12

integrated i18n into sicp repository

e82194a

Add .env to .gitignore

7e793de

attempt to create a translate function that attempts to recursively s…

098de27

…plits the text into manageable chunk

Refactor translate function to improve XML error handling and update …

10856a1

…output path

Update .gitignore and refactor translation logic to use recurTranslat…

4922e98

…e function

updated splitting logic to concat orphaned texts

9f21e76

Refactor XML parsing logic to use strict mode and enhance text escaping

29f1441

Fix error handling in recursivelyTranslate function and remove debug log

f0316dc

Refactor recursion logic in translate function to improve orphaned te…

e45ede2

…xt handling and error reporting

Enhance translation performance by measuring execution time and refac…

ba88b63

…toring segment handling to use arrays for better management

Refactor async execution in fancyName function and improve error hand…

d443ffb

…ling in Promise.all

Refactor XML escaping logic to use strongEscapeXML for improved secur…

da52eb5

…ity and consistency

Refactor segment handling in recursivelyTranslate function to improve…

3370e50

… merging logic and remove unnecessary debug logs

Add .gitignore for node_modules and update translation logic to handl…

d48e425

…e TRANSLATE tags

update lockfile

806a152

reorganised file structure and yarn prepare json to translate all lan…

3329c54

…guages in xml

Merge branch 'master' of https://github.com/yihao03/sicp

7a835c5

Fix file path resolution in translate function and update package man…

d110429

…ager version in package.json

Refactor translation logic to simplify segment handling and improve r…

a115312

…eadability

Enhance error handling and logging in translation process, streamline…

89841d6

… file path usage

added proper error logging and reporting. added ability to skip trans…

b37b503

…lations inside SCHEME and SCHEMEINLINE tags

modified index.js to properly handle updated repository structure. Ad…

48fe8df

…ded try catch when parsing xml to json to report failed parsing possibly attributed to unsound xml structure.

added troubleshoot toggle that skips calling

8e0e318

openai api

Refactor translation logic to improve handling of SCHEME and SCHEMEIN…

b9ce30a

…LINE tags in XML parsing

added filename to json error logging

5f23def

updated to remove file search annotation generated by api calls

429cff5

yihao03 added 2 commits April 12, 2025 15:13

ran prettier

a28c5f8

refined troubleshoot toggle, temporarily removed problemtic skipping …

c3be2c0

…logic

renamed files

cdf91c6

yihao03 requested a review from arnav-goel10 May 2, 2025 05:02

coder114514 added 4 commits May 7, 2025 23:54

Merge branch 'sicp_i18n' into i18n_gh_actions

d0c95f7

index.js: only write search data for English, stop other langs from b…

fbf8f6a

…reaking search data

index.js: remove import from ../i18n/config

4bbb406

format

bbe00bd

arnav-goel10 requested changes May 15, 2025

View reviewed changes

coder114514 added 8 commits May 30, 2025 18:03

remove files that are in the wrong place

f3eb480

translated section2 cn xmls

remove the npm lockfile in i18n dir

ebeed10

ignore the logs in i18n dir

3d0d5e3

update .env in workflows for translation

7ec10f1

add packageManager and license part in package.json of i18n

6f0c85c

remove unused module according to code review

a53b54d

replace MAXLEN with max_chunk_len according to code review

4125ea4

add definition of __dirname to processHandler.ts

626b0d2

coder114514 requested a review from arnav-goel10 June 5, 2025 14:03

coder114514 self-assigned this Jun 5, 2025

fix typo in the workflow translate-everything.yml

71395f4

arnav-goel10 approved these changes Jun 6, 2025

View reviewed changes

yihao03 and others added 2 commits June 10, 2025 22:36

Merge branch 'master' into master

1d4307e

Merge branch 'master' into master

a5c1a03

RichDom2185 requested changes Jul 5, 2025

View reviewed changes

fix file perms

92d1ce1

fix accidental change

af25668

remove zh_CN directory

ea0351a

		echo API_KEY=${{ secrets.OPENAI_KEY }} >> .env
		# echo API_KEY=${{ secrets.OPENAI_KEY2 }} >> .env

Added i18n component and related scripts #1082

Are you sure you want to change the base?

Added i18n component and related scripts #1082

Uh oh!

Conversation

yihao03 commented Mar 28, 2025

Uh oh!

yihao03 commented Apr 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnav-goel10 left a comment

Choose a reason for hiding this comment

Uh oh!

RichDom2185 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coder114514 Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coder114514 commented Jul 5, 2025

Uh oh!

RichDom2185 commented Jul 5, 2025

Uh oh!

coder114514 commented Jul 5, 2025

Uh oh!

RichDom2185 commented Jul 5, 2025

Uh oh!

coder114514 commented Jul 5, 2025

Uh oh!

Uh oh!

coder114514 Jul 5, 2025 •

edited

Loading